Papers with grounding natural language instructions
Mapping Natural Language Instructions to Mobile UI Action Sequences (2020.acl-main)
Copied to clipboard
| Challenge: | a new problem of grounding natural language instructions to mobile UI actions is emerging . we use a Transformer to extract action phrase tuples from long-range natural language instruction . |
| Approach: | They propose a dataset that pairs English instructions with actions performed by people on a mobile UI emulator. |
| Outcome: | The proposed model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PixelHelp. |
StructuThink: Reasoning with Task Transition Knowledge for Autonomous LLM-Based Agents (2025.findings-emnlp)
Copied to clipboard
| Challenge: | StructuThink framework enhances LLMs' ability to ground decisions in domain-specific scenarios. |
| Approach: | They propose a knowledge-structured reasoning framework that enhances LLM-based agents with explicit decision constraints. |
| Outcome: | The proposed framework achieves higher task success rates and more efficient action sequences than baseline methods. |